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ABSTRACT 



Association for 
collaborated on 



Piftaan member countries of tha Intarnational 
tha Evaluation of Educational Achievement ' IEA) 
a project that had four aims; ID to contribute to 
the conceptualization o*;the domain of writing, (3) to develop an 
internationally appropriate set of writing teaks and a system for 
assessing compositions, (3) to describe recent developments and the 
current state of instruction in written <com£>ait ion, and (-4) to', 
identify factor?) that explain differences and patterns in the 
performance of written composition .\A set of writing 'tasks was 
presented in each country to one' or more of the following population 
groups: <i) the end of primary education (Population A), (2) the end 
of compulsory education (Population B), and (3) the preuniveraity 
year (Population C). The recommended sample sizes were 50 classes 
Populetions A and C, and 100 classes for Population B. Problems 
relating to the construction of writing tasks, the, allocation .of 
writing tasks, and the scoring of student scripts were addressed, 
most appropriate scoring system combined a holistic,'; * 
overall-impression .marking and an analytical marking. It was 
concluded that in spite of a great effort*the scoring metric waa too 
elastic to allow robust cross-national comparisons of scoraa on any 
one task or group of tasks. A sample scoring sheet, tables; and 
diagrams are included. (DF) * . * 
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' INTRODUCTION to the educational RESOURCES 

INFORMATION CENTER (ERIC)." 

In this paper we will give a brie-f account g-f an c>-going 
'international 'study of written^ composition. Fifteerf .member 

< • • :: 

countries' of the IEA (The International Association for the 
Evaluation of Educational Achievement) from all over, the' world f 
are collaborating on the project. A fairly^ large set of writing' 

tasks has been* presented in some countries to one population of 

\ ■ 
• \ * 

students, in some to two. and in some to all three populations, 

% mm 

representing end of primary educati.bn (Pop. A), end of compulsory 

* * • 

. education (Pop B) , and thejDre-uni versi ty year . (Pop C) , respecti- 

vely (for: a detailed account, see Table 1, Appendix 1). The 

recommended minimum sample sizes were 50 classes for Populations 

A and C and" 100 for Pdto B. 

Me will .first place the study within its larger context. 

After that we proceed to describe the aims of the project, its 

desixjn and scope. . We will, give particular emphasis* to the taskte 

and to the scoring procedures. We, will al'so present some results , 4< 

concerning the pilot stages o+ the project, 

*• • 
We wish to point out that this, is not the "official 11 

4 

or ^complete presentation of the IEA Written Composition Study- We 
vhave chosen to highlight some aspects of the study and have added 
some personal reflections and interpretations, 

v. * ' 

. Paper pj^?sented at an International Writing Convention, School • 
o-f .Education, University of 1 " East Anglia, Norwich, March 31-April 



,* -. 



• (JONTEXT QF .tHE STUDY \. \ 

Since the late 1950 's, a number of educational researcher^ 
and research institutions have been working on an empirically 
'oriented comparative research ' program. A small -feasi(b»i 1 i ty 'study 
was carried out by £he IEA in 1959-1962. This was followed by a 
First Mathematics S^udy in the mid 1*3(60 's (with 12 countries/ 
school systems participating) and by sfix studies in 1970-1971. 
These covered Science _(19 countries}, Reading Comprehension % <15), 

Literature (10), French as a foreign language "(8), English as a 

— ■ ■ /• • 

forejfgn , language*. ( 10) , and Civic Education dp). On-going studies 

comprise a sec6nd mathematics and science study and a study of 

classroom activities. Since 1979, work has been jcrarried out on an 

international study of written composition. ^ 

A number of international and national reports have been 

published on the completed studies. Similarly, a great number of 

journal articles have been published, as a bibliography of IEA- 

related publications shows. 



"1 MANAGEMENT THE STUDY 

Like all IEA studies, the Written Compositions study has' a 
complex management otj-ucture. The International Project Council 
at its annual meeting makes general policy decisions. Dr. Alan C. 



Purves from the University of Illinois at Urbana-Champaign is the 
Chair of the* IPC* More speci f ic planning is the responsibility of 
the International Steering Committee, chaired by Anneli V&hapassi 
from Finland. Members are Tom Gorman (England & Wales)., Judit 
Kradar-Fulop ^ (Hurvgary) , Eva Baker (USA), Alan C. Purves (USA), 
Hildo Wesdorp tNether 1 ands , until 1935)^ Pai Obanya (Nigeria) and 

VtaiflKj Konttxnen (Finland). SauJ>i Takala (Finland) is the Interna- 

• » 

tional Coordinator and Elaine Degenhart (USA) Deputy Coordinator. 
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From January 1981 toVfall 1984, the International Coordinating 
Center was located at the Curriculum Laboratory .(UIUCX. Since the 
.fall of 1984 the coordinating of -the project is managed 
by the Curriculum Laboratory and the Institute for Educational 
Research ^ University of Jyvaskyla. « • 

National Research Coordinator^ constitute the International 
Study. Committee, ; which has .met a few times to discuss* the 

■« ■ i 

0 

% 

implementation ,of the study according to common plans. v 

The costs of international coordination have been paid by/ 

the IEA, while the national costs of implementing the study are 

- ^ 

paid by each participating country. « N 



SIGN. AND AIMS OF THE STUDY 



The IEA International Stu^ly of Written Composition seeks to 
accomplish the following tasksjs • . 

(1) to contribute to the conceptual^ zatron of the^ domain * of 
writing and particularly the domain of school-based written 

-composition, 

(2) 'to develop a an internationally appropriate set of writing 
tasks and a system for assessing compositions whith is 
appl i cabl e across countries/school systems and across 
languages, 

(3) to describe retent developments and the, current state of 
instruction in written .composition in the participating 
countries/school/ systems, and 

(4) to identify factors which explain differences and .patterns 
in the performahce of written composition* and other 
outcomes, with particular attention to cultural background, 
curriculum and teaching practices. 

The independent and dependent variables of the stud/ and their 

presumed relationships are illustrated in Figure , ,1. We will 



Place Figure 1 about here 
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\ • ' . *■ ■ •■ • • " 

begin with the conceptualization and' operational isati on of the 
dependent variable. . 

. DEPENDENT VARIABLE: WRITING TASKS AND SCORES , ' 

\ . ... . • 

» . . . * .« 



- a,- n«. 



X 



Problems related £q tfte deeegdent variable 
the greatest, challenge in the study. has been the work on the 
dependent variable. Here we can give only a brief account of it 

(far a detailed account, see Vahapassi , 1982, 19S3). Three 

• « * 

problem areas had to be addressed: ,. 

It Pr oblems related to the* construction of writing tasks ■• l\ 



4 ». 



(a) What is the total domain of writing,, especially school- 
* "based domain of written composition? This, required 
conceptual analysis and 'synthesis. 



0 



fa' ' 



(b) What is the appropriate sample from the total domain' -f 

for. the students-concerned? What sub-domains should be- .• 

included in the set of writing tasks? * '." 

— . ' »• 

<c) What is*/the appropriate system, for specifying the • % 
tasks? 



(d) How should the actual writing ta^ks be formulated? 
II. Problems related to the allocation of writing tasks 



* 



^ <e) What kind of tasks should be presented to each of the > y i 

three student populations? . « i . 7> * A r 4 

.♦ . ■* ' % 

<f) How should populations be linked- through common tasks? 

<g) How many tasks should/car? each student be asked to % 
respond to? - - $ 

(i) If task rotation is necessary, how should it be done v v 1 

so as to maximize the information obtained and . , 

; minimize problems that are. related to rotation? % 

■ * ^ 

III. Problems related to rating of student scripts \ 

*M ■ 

I < ■ i|i 

<j) What rating system should be used (e.g., holistic., 

primary trait, analytical)? \ > 

( k': ) How can a rating system be constructed which t can be 
applied in a comparable way in all * participating 
countries? 

It is possible-' to focus here on a few of these problems only. 



;.?f 
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After an extensive review of the problem area a/id after 

• 4 " . . - , ' 

having tried several systems and discarded* them, /the . following 

■ . :<> ,' * . ' 

approach (Figure 2) 'was adopted (for a more* detailed account ; see 
VahSpassi, 1982, 19831. , , .. . I 



BaasBSBsssssasaessBsiisana \ 

Place Figure 2 here . * 



The system relates, the primary purpose and audience -of writihg/^o 
the .level and object of cognitive processed involved, in the 



writing task. This gives us a» typology and we can test it in some 



respects hy trying to place various communication tasks, and 

writing products in i*ts cells. This has been illustrated 

selectively in Figure 2. Without " going into detail \ suffice it to 

say that up to now, the model has serve'd as a very useful tool in 

solving a number £i problems. We will return. to the model later 

r 

T" 

an ; • - . * 1 

SsmE.Li.ng, of tasks from, the domain. 
In the l£A study, writing has been considered an act of 
communication about some topic, betwee^""\he writer and the 
reader (s) (addressee/audience). The writer has a certain purpose 
in writing. and his ultimate goal is communicative effectiveness 

♦ - 

(effective expression) .\ Typically the writer, produces a. text 
alone, without the immediate feedback from (cooperation/negotia- 
tion with) the reader.* This latter—aspect seems to be one of the 
mast important features that distinguishes writing from face-to- 

■ 

face conversational interaction (Takala, 19B2, . 1983). The above 
discussion has been shdWn schematically in Figure 3. 

Place Figure 3 here a 
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The ' tasks of • the IEA W*itten ; Composition Study' can be 
r-elated to/ .ie domain specification grid (Figure 2) and to the 
rhetorical model of writing (Figure 3) as follows: * * • . N 

» • » A ft » 

/U) Tasks that emphasise the perspective of the writer. 

Ia§k 5» Jtor ite a personal' stpry • 1 ' 

lIIE §» "write a "free" composition on an ambiguous and 
• . evocative pictorial stimulus ; % & . 



s 



<2) Tasks that emphasize the perspective of, the topic 

Task 2: summarize %i text y ■ 

I^Ik.i: '..retell a story (in a shorter form) 
Task 4a: describe a ritual mask .#""*' 
iSk 4b: describe a process of doing something 
I3E write a reflective essay 

(3) .Tasks that emphasize* the perspective. of the reader 

Task 6:' try to persuade the reader to share the writer's 
0 strong view about something ' . 

•<4) Tasks that have several perspectives 

« » 

... • " * ' • 

lask la.: describe a desired bicycle to an uncle who- 

. ~" wishes to buy one.as-^ birthday or esent 

I«sk lb: describe oneself to a. penfriend whom the 
studgnt is going to visit so as to make 
it possible for the penfriend to identify 
the student as he comes to meet the student 

». 

Task'lc: write, a note to the principal /headmaster 
cahceling*a scheduled meeting 

Task Id: leave a message, at home lling where the 
student has gone after school 



t • 



I^sk le: write? a letter applying -for an advertised • 

summer job . * 

• a 

Note tnat tfie writer — reader social status anrii the topic is 
varied s/stematic:al 1 y . * • 

V 

T.ask 9: write a letter to a younger student who is 
coming to study at the same school as the 
t writer, telling the new student how he/she 
shquld write in the new school "to get good 
grades. 

If we focus on the purpose of the tasks, we can see that 
Tasr% 1,.. 2, 3, 4 and 9 have a predominantly informational 
purpose. The ©xpressive purpose Is dominant in Tasks 5 and 8. 

' BF«tmPYAV "UfiLI 



Tasks le and 6 have a. per suasi ve, , puime^ ; % Jask 7 has an 



■^111 



'•fir 



e:<pl icati ve/interpreti ve. purpose 

If we pay attention to the cogni tve' ; structu «■ of the, tasks, ^ 
we might suggest that Tasks ic, id, 2» 1 3, J.- S have a 

v struct'ure brised mainly on a. temporal organisation (an account of 
, eveqts). Tasks la, lb and 4a have a structure based mainly on a 
spatial organization (an account of the physical characteristics 
of objects). Tasks le, 6.* 7, 8 and 9 have a *tru£|ig$* based 
mainly on a logical organisation (an* account of ideas and. thought 
structures) . Other classifications are, o* 'court's, possible. For 
an example of actual tasks, see Append-ix 2. s ; 

. Scoring system _ 

Scaring scheme 

* 

After considering and ^r.ying out various scoring^ systems 9 

^ \ ' « 

^holistic, primary trait i analytical), it was agreed that the 

most appropriate one was a system which combined a holistic 

« « 

overall impression marking^ and analytical, marking. Figure 4 is 
. an attempt tq summarize the conceptualization work done in deve- 
loping the* scoring system. *v 



■\t>- 

i 



i 
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I 



Place Figure 4 "here ( . ? 

ssssB^ftjrSsxsssssssssssssssiwwKssHSSSssssBSsKsj »■ ^ ' £ 

'■ * ' .'* 

The overall construct "student writing" is divided into 'two 
sub-construct^: "writing, competence" and "writing preferences". ^ 
'We are cdrfcePfied here mainly with writing competence, which is 

* ' \ . r 

♦ - ; .... 

taken to consist' of. "discourse-structuring competence" (or 

* "? 
rhetorical cqJpetehce) and of "text-producing competence". 

.•t 

« * " V 

Rhetorical competence is assumed to require both "cognitive 
competence" and "social competence". 

> . BEST COPY AVAILABLE 
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. . . « . - " ■ .» ,-.-.-»■"."•. 1 • ■ ■ ' ( ■ - —.-.-Ira*- 

" ' ' ••' * ! * 

- Cognitive competence is manifested .in illotrutiohary effecti- 

*.■■"• * , a 

!':;:"/ veness, i ..e. ^jn the ability of writer* to make fMdri easily N 
recognise the cc ...muni cat ive intent. TKis is operational! zed. in * 
our scoring scheme as rated "Quality and scope of ideas" and 
as rated success in the ^'Organization and 'Presentation of Con- . 
tent". In some tasks, writers also aim at perlocutionary effecti- 
veness, i.e., they want to get the readers to do something. This 
may mean that writers wish to have readers entertained 'or 'have 
♦ them change their minds,, or even do .something concrete .after such 

a .Chang's of mind. Such perlocutionary effectiveness presupposes 

awarenes gf readers' norms, attitudes, etc. In our scoring scheme 

• . \?. ... 

< - • ■ ■ • 

this is operationalized.in part, aj "Style and Appropriateness of 

; - Tone"; ' * . '. \ '% 

Text-producing \ competence presupposes "linguistic compe- % 

tence" and "motor competence". These ar£ operational i zed in our Sf 

• ■ ! ■ ..... 

* * "V*;:: 

•scoring scheme o as rated competence in using "grammatical fear "M 

tures", ^'spelling and ortographic conventions", and "handwriting : : | 

and neatness", all rated in accordance with national norms. , For a |f 
sample scoring scheme, see Appendix 3. :| 

The fact that raters are readers and will, in spite of $ 

i ■ ; l 

their attempt " to rate compositions as objectively as possible, 

* * .t 

according to tge categories of the scoring scheme, have a 

subjective response to the scripts. For this reason,, we have 

added a category, "Response of. Rater % which allows raters to 

» # ■ ■ 

- . indicate such a response. This is another way of trying %o get an 

estimate of perlocutionary effectiveness. t ; 

Scoring scal.es ' 

t^he scoring scheme is supplemented* with a scoring scale, 
which consists of a number* of compositions that exemplify various 
scale point values of the scheme. There were no particular , 1 

o problems in having a consensus on the adopted scoring scheme. On , 

iSJC • " g . BEST COW AVAILABLE | 



the other hand, the construction of the scoring scales was a 
difficult and laborious task. ■ 1 

There were several problems that had to be addressed. The 

♦ 

most difficult one wa* how to establish internationally agreed 
scores f or compojsi tiona wri ttt.i in several different languages. A 
number of alternatives were considered (e.g., translation into 
several^ ^languages, use of bilingual raters, having an 
international rating panel). After long discussions, it was 
agreed -that the most feasible .solution was as follows! <1) Pilot 
test the tasks*. (2) National Centers select a Set of compositions 
representing the whole performance ra/ige, and rate feh^m according 
to national norms. <3) Translate these proposed "benchmark" com- 
positions carefully into English, following a common ! set of 

guidelines. <4). Assemble juries, with an international composition 

0 .* .... 

and have them rate the (sometimes translated) compositions after 

the juries have been trained .according to the established train- 
ing system. (5) In rating the compositions of the main st^udy, use 
the original benchmarks (written in the source language) with 
their international scores in rater training and as a reference 
in final rating. (6) Check the comparability of rating in parti- 
cipating countries by means of a common "international calibra- 
tion set", which is a set of compositions with a high degree of 

V i • 

cpnsensus, 

« « 

Some Results 

The system described in the above was applied for the first 
time during the third International Study Committee Meeting in 
Urbana in the spring of 19B4. . 

Four international juries consisting of five National 
Research Coordinators were set up. Members of the Steering Com- 
mittee served as jury leaders and natiye speakers of English as 

• BEST. COPY AVAILABLE 



rapporteurs, ' 




The establishment >*rbenchmarks and calibration sets was , _; : f§ 
limited mainly to Population 8. Altogether some 1,400 



compositions had to be rated, discussed, and commented on. -Each 



)m*pesitTohs. 



jury member^scored about 500 compositions. Jury members knew only 

i *■.■*•-' . ' 



the target aspect that had been given, * .nationally proposed 

initial score. The compositions were handled in the following 

ways , - • ^ 

* - if there was a 4 out of 5 agreement on ^ the nationally 
proposed target aspect, the score was international validated 
(whether it was identical with or lower or higher than the 
national suggestion) . - - 

-if there, was % .lower agreement - or if a jury wantgd another • 

a look at some compositions for whatever 
compositions were reierred to rescoring 
tions" received, highly uniform scores on all 
target .Aspects, they were singled out as candidates for. the 
international qitllbratican • • * • • 

-« \ ' ' ' v * . : 'f 

The results of \ the Urbana scoring session have been partly *g 



jury to .Have 
reasons '*- such 
if some composi 



i + ■ 



analyzed and can be briefly summarized as followss . * 

' • I j • • * 

(1) By using a common set'6f compositions scored by all jury' 

- ■ ' • ' ! ' • 

members after they have finished scoring a particular task, it 
was found that there was full agreemenjb in 22'/. of - all cases, 
78,27. agreement with one scale point deviance. • This was lower r- 
than we had hoped, but not unusual eyen when raters belong to the 
same interpretive commu tity. The produced benchmark scores do not 
constitute as good a yardstick as is desirable. 

(2) There was some systematic difference in jury leniency ( mean 

t 

f of all differences in means between all pairs of juries .326). £ / 

• / 
(.3) The average standard deviations of juries varied only / 

? 

slightly. , r .. ^ 

( 

(4) The perception of quality of compositions was to a large 

• # -\ 

9 

extent convergent between 5uries as median correlations ranging 
between .83 and .91 on different aspects show. . • . * 

.., • - • - . 

, <5) Jury homogeneity was fairly good (.85 or higher, Cronbach's 

ER |c : - ' \ „ BEST COPY AVAILABLE 
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(6) There was considerable scale shrinkages the 5-point scale 
shrank to a-3-poinrt scale in about half -of-tha cases. This; is to 



some, extertt an artifact, because in some cases countries had a \ 



.small , pi lot sample and reported that they sometimes felt obliged 

to propose a 5 Bven if they felt that that was too high. 

• " ' ' ' - ' .* 

(77 More problematic are the ijO?. of all cases .wJien the 

international jury reversed the scale (e.g., gi ving«.Z-5-4 br 3-2- 



4 when the proposed order of merit was 1-3-5). 

, The results of the second searing jneeting held in FTrascati 



late 1984 have-not been fully analysed yet. 



^ The implications of , the results were discussed at a meeting. 
i,n Hamburg iV» February, 1985. It was concluded that in spite of a • 
great effort' the. scoring metric /is too elastic to allow j robust 



cross-national comparisons' of/ scares' on any one or group^ of 

*' * •'/■ ' ' 

tasks. -Since there is quite a lot of interest in some 

° — 

' ■ *», 
international comparison, some further .work is, /leeded. There are 

plans # , to 'try to improve the comparability using one task and one 

^population only. A robust international scale is created and a 

subsample of compositions are rescored. *■» This 'one task will then 

be used for limited international Comparisons. / 

■ . X 

For anyone working in the assessment of written cdVnposition 
the resijltsyqp an international attempt' to create a -fully compa- 
rable scoring procedure . are not surprising. In fact, we have^ bgen 
somewhat surprised that we have been able to solve as many 
problems as is the casp While robust comparisons of achievement 
across countries are doubtful, we do not consider that we have 
failed in our attempt. It appears that tasks 1, 2, and 3 (see 
Design and Alms of the Study above) will be successfully 
accomplished. That is more chan some df us dared to expect. 
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INDEPENDENT VARIABLES 

\ 



Figdre« . 1 presents t;he major constructs that underly the 



large set of variables used as independent variables* Derailed 

.... • . 1 - ; - \ 

information ha* been gathered by questionnaires, interviews aind 

written case studies about the context of written composition 

teaching. , - 1 

The gathered information (covering hundreds of 'variables) 

wi}l be used mainly to, provide detailed descriptions of the 

context and status of ' writing instruction in participating coun- 

'4 . • 

• ' . . *• 

tries. A smaller subset of variables will probably be used in 
some multivariate analysed and model testing. 

Me are particularly t interns ^ in the portraits o^ writing 

instruction that the very speci Teacher , Questionnaire data 

\ • ■ ■ . "*■ v v 

will provide. \ 

REPORT I NB OF RESULTS 
' c '' There w$l 1 be both international ^nd national reporting on 
the findings of . the study. Three international reports are 
planned. The first volume will deal with, the dependent Variable. 

, s * 

It wi41 give a detailed account* of the problems and issues in*- 
volved in constructing an international set of writing tasks and 
i n aco^L ng s tudent scripts using an international ly agreed -i^pon 
sroring procedure. The repbrt is ^expected to be in manuscript 
form by the end of 19B5. 

The second volume will give a detailed description of the 
context and practices of writing instruction in the participating 
countries. . . 

{J, The third volume will present the main results of the study. 
This will probably be a set of parallel national • portrai Is with 
some international comparisons. The extent of comparisons will 

. BEST COPY AVAILABLE, 
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DIP'-- : - • ; .. ' ■ — w ' ^ ' 

depend on the degree to which the scoring is, in fact, comparable «:^ s ; 



\ 



in the participating countries. 



SOME CONCLUDING REMARKS • ' 

* * ■ 

In conclusion, we would like to make some personal 
observatio.is. The IEA writing study has .been the most difficult 
arid laborious task ever undertaken by either of us. It has forced 
us to tackle many/ problems without being able to draw on any gopd 
models. The pressure of deadlines has also been very, heavy. ' 

le other- hand, the IEA study has been more rewarding 



than 




other project we have been working on. ' Facing 

* *» 



new 



an 



problems . has been^ a ^cdnstant^chal lenge and given us 
opportunity to learn a lot. The chance to work on our problems 
with dedicated and experienced colleagues from all over the worLd^ 
has been invaluable in -finding justified and acceptable 



solution^.- It 



has-Blso taught us to be more sensitive to cultural 
similarities and differences. We have come to be more aware of 



\ 



the variety of ways in which writing instruction can be arranged. 

\ On the basis of the enthusiastic reception the IEA study hits 

had in our country , we are convinced that the project will have 

a major impact on the teaching of qomposi tions in Finland. 

Through our participation in the IEA study, we in Finland will 

j 

get a first good national /assessment of writing. In addition to 



that, we will be able to relate our situation to that in several 

s i 

other countries. We have established close contacts with many 
col leagues in other countries. 

There are plans to store a representative sample of student 
scripts in an international student text corpus. Several signs 
indicate that there is growing interest to move from the 
assessment stage t» a stage, when we can take a close look at 
the compositions themselves. 'A number of cognitive, linguistic, 



..... j. . 
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rhetorical.!; crosa-cul tural etc studies have been " tentatively 

r . ' 

sketched during our recent meetings. * 
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Sample Scoring Sheet 
TASK 6 ARaJMENTATIVE/PWSUASIYCtOMPOSITIOH 
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